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Abstract. We introduce a modification to the patchy method of Navasca and Krener for solving 
the stationary Hamilton Jacobi Bellman equation. The numerical solution that we generate is a set 
of polynomials that approximate the optimal cost and optimal control on a partition of the state 
space. We derive an error bound for our numerical method under the assumption that the optimal 
cost is a smooth strict Lyupanov function. The error bound is valid when the number of subsets in 
the partition is not too large. 
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1. Introduction. In this paper, we introduce a numerical scheme for solving 
the infinite horizon optimal control problem of minimizing the integral 

l{x,u)dt (1.1a) 

of a Lagrangian l{x,u) subject to the controlled dynamics 

i = f{x,u) x{0)^x° (1.1b) 



where / and I are both smooth, and I is strictly convex in the control u e M™ for all 
states X G E". The solution that we are interested in is an optimal cost tt : M" -> M, 
which is the minimum value of (1.1a) incurred by driving the state from an initial 
value of X to the origin. The optimal control that drives the state to the origin is given 
in feedback form by u = k{x). Our solution is a set of polynomials that approximate 
the optimal cost and optimal control on a partition of a subset of the state space. 

Our method is a modification of the original patchy method [7 , which we altered 
to obtain the error bound in Theorem |3.6| Like the original patchy method, it is an 
extension of the Cauchy-Kovalcvskaya method [4 , the fast marching method 15]. 
the patchy technique of Ancona and Bressan [2 , and Al'brckht's power series method 

Al'brekht's method is an algorithm for computing a series solution to the optimal 
cost that must be centered at the origin of the state space. A drawback to the 
requirement that the series solution be centered at the origin is, even when the optimal 
cost and optimal control are smooth over the entire domain of interest, their power 
series solutions may be local in nature. This means that outside of some region 
containing the origin, it may be impractical to compute the number of terms necessary 
to achieve a desired accuracy. The patchy method that we present uses Al'brekht's 
series solution as an initialization step, and generates a new series solution centered 
at a point away from the origin. Doing this enlarges the region where we have a valid 
approximation to the optimal cost and optimal control. 
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1.1. The Hamilton-Jacobi-Bellman PDE. It is widely known that if the 



optimal control problem ( 1.1 ) has a smooth optimal cost n{x), and the optimal control 
can be put in feedback form u — then the optimal control and optimal cost 

satisfy the Hamilton-Jacobi-Bellman (HJB) equation 

= min(^{x)f{x,u) +l{x,u)) 

" (12) 
k{x) = argmm [^-{x)-{x,u) + -(a:,«)j 

locally around the origin. Furthermore, if + l{x,u) is strictly convex in 

u locally around a; = 0, u = 0, then the HJB PDEs ( |1.2[ ) simplify to 

Q= -^{x)f{x,n{x))+l{x,K{x)) (1.3a) 

= + (l-3b) 

1.2. Solutions to the Hamilton-Jacobi-Bellman PDE. The existence of 
a solution of the optimal control problem ( |1.1[ ) around the origin is determined by 
the leading order terms of the problem data / and I. Suppose the dynamics and 
Lagrangian have Taylor expansions 



x = Fx + Gu + J2 f^^^ 

k=2 

^ oo ^ ^ 

l[x, u) = - {x'^Qx + u^Ru) + ^ /[''I {x, u) 

fe=3 

where I'^l denotes degree k terms in the power series, R is positive definite, and Q 
is positive semidefinite. The optimal control problem is said to be nice if {F, G) is 
stabilizable and {Q'^^'^,F) is controllable. 



Al'brekht [1 showed that when the optimal control problem (1.3) is nice, the 
optimal cost and optimal control have series solution centered at the origin of the 
state space. Lukes [5] showed that both these series solutions converge to the optimal 
cost and control under suitable conditions. Al'brekht 's method is an algorithm for 
computing both these series. It works by computing the degree j and j — 1 terms as a 
pair in the expansions of the optimal cost and optimal control, starting at j = 2 and 
terminating at some user specified degree. The original description of the algorithm 
is in Al'brekht 's original work [1], and a generalization with notation consistent with 
this paper is in [7]. The Matlab implementation of Al'brekht 's method that we use 
in the patchy method is available by request [S] . 

2. Derivation of the patchy method. 

2.1. Optimal control problem assumptions. We will derive the patchy method| 
for the special case of the optimal control problem where the control is scalar (m = 1), 
and the dynamics and lagrangian have the form 

x = f{x)+g{x)u l{x,u) — q{x) + -r{x)u^ (2.1) 



where 
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/, 5 : M" ^ M" q,r : M" ^ M m G 
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The patchy method does not require these assumptions, but they do significantly 
simphfy its derivation. Furthermore, we assume 

1. /, g, q, r have continuous partial derivatives of to degree d + 2 on Xc- 

2. The optimal control problem is "nice", meaning it can be expressed in the 
form of (1.4), and {F,G) is stabilizable and (F, Q^/'^) is detectable. We also 



assume that Q is strictly positive definite. 

3. TT has continuous partial derivatives up to order c? + 2 on 

4. g > and r > on 

5. The optimal cost is a strict Lyapunov function on Xc^ meaning 

^{x)x = ^ix){f{x)+g{xMx)) < (2.2) 
for all x in \ {0}. 

Assumptions [T] and [2] guarantee that assumption [3] holds in some neighborhood of the 
origin. The Taylor series of tt in this neighborhood can be computed by Al'brekht's 
method. The fifth assumption is essential in both the derivation of the patchy method 
and the proof of its error bound in Theorem |3.6| 



2.2. Overview of the patchy method. Although the method works in higher 
dimensions, we will assume in the overview that the state space is two dimensional. 
This allows us to illustrate how we partition the state space. We first compute the 
degree d + 1 Taylor approximation to the optimal cost at the origin by Al'brekht's 
method, which we denote tt". We assume that all truncated series solutions to the 
optimal cost and optimal control are polynomials of degree d + 1 and d. We then 
pick a sublevel set 7r° < c on which tt" has an acceptable level of error. We refer 
to this sublevel set as the Al'brekht patch and denote it as A. The patchy method 
expands the domain of the numerical solution to a superset of the Al'brekht patch, 
which we refer to as the computational domain and denote it as Xc- The patchy 
method picks a patch point, denoted x*, on the boundary of the Al'brekht patch and 
then computes an approximating polynomial to the optimal cost that is centered at 
the new patch point, which we denote as 7r\ The core of the patchy algorithm is a 
method to compute the approximate partial derivatives of the optimal cost at x*, thus 
computing an approximation to the Taylor polynomial centered at x^ of the optimal 
cost. The patchy method repeats this process until the origin is surrounded with a 
number of patch points. 

The computational domain is partitioned into a set of patches during the course 
of the patchy algorithm. Each patch point is associated with a single patch, and 
the optimal cost and optimal control at any state that falls inside in the patch are 
computed by the approximating polynomials centered at the associated patch point. 
Every point in the computational domain is associated with exactly one patch, and we 
compute the approximate optimal cost and optimal control at the point by the series 
solutions centered at the associated patch point. In two dimensions, the boundaries 
of the patch are the boundary of the Al'brekht patch, a level curve of the optimal 
cost centered at the patch point, and two lateral boundaries that intersect the other 
two boundaries. Illustrations of the Al'brekht patch and a typical patch in a two 



dimensional state space are included in Figure 2.1 

Once the Patchy method has encircled the Al'brekht patch with new patches, it 
generates a new ring of patches and associated optimal cost polynomials, where the 
new patches form a ring that encloses the old set of patches. 
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As we will see, under some standard assumptions on the underlying optimal 
control problem, as well as the assumptions that the true solution is both sufficiently 
smooth and a strict Lyapunov function, we can derive error estimates on the patchy 
method. Unfortunately, these error estimates do not go to zero as the density of 
patches on the computational domain goes to infinity. To our knowledge, the only 
algorithm that provably solves the nonlinear Hamilton-Jacobi-Bellman equations in 
multiple dimensions with higher order accuracy is the finite difference scheme due to 
Szpiro and Dupuis [H]. Their method is provably second order accurate on subsets 
of the computational domain where the true solution is smooth. 

2.3. Placing a new patch point. Given the previous patch point and its 
associated approximate optimal cost tt*, the patchy method must determine where 
in the state space to place the new patch point a;*+^. If the computed optimal cost 
TT* is sufficiently close to the true optimal cost in a neighborhood of a;*, then by 
the assumption that the optimal cost is a strict Lyapunov function, the closed loop 
dynamics is not tangent to a level curve of tt* in the neighborhood. Thus the partial 
derivatives of the optimal cost can be calculated by the technique of this section at a 
point on an appropriately chosen level curve of tt*. The patchy method places x^~^^ 
on a level curve of tt* under the constraint that the distance between a;' and x'^'^^ is 
less than h, the maximum consecutive patch point distance. We derive the maximum 
consecutive patch point distance in chapters [4] and [sj 

2.4. Computing the optimal cost at a new patch point. Away from the 
origin, the HJB equations alone do not fully specify all the partial derivatives of the 
optimal cost at a given patch point. As a consequence, the patchy algorithm computes 
some partial derivatives of the optimal cost from the HJB equations, and computes 
the remaining partial derivatives from the previous optimal cost. 

The computed optimal cost at the new patch point x'"*"^ is calculated by inheri- 
tance, meaning we set 

2.5. Computing the first order partial derivatives of the optimal cost 
away from the origin. First, we compute n, the gradient direction of 7r'+^ at 
from tt' by 

' '""'i^-^)- (2.3) 



111^(^^+^)11 dx 

If the level sets of the previous and new optimal costs are tangent, then the gradients 

of the previous and new optimal cost are coUinear at the current patch point. In this 
case, there is some positive scalar z such that 

^^(x*+i) = zn (2.4) 

To compute we first define the scalars 

/„ = n./(.T*+i) and = n ■ g{x'+^). (2.5) 



We can derive a formula for the optimal control in terms of the gradient of the optimal 
cost by solving (1.3b I under the assumption (2.1). If we substitute this formula for 



the optimal control into (1.3a I, then it reduces to the scalar quadratic equation 

r,2 







2r 



^-^z +/„z + q(x+ ) 



(2.6) 



under (2.4 1 and (2.5). If the computed and exact gradients of the optimal cost are 
sufficiently close, then it follows from the assumption that the exact optimal cost is 
a strict Lyapunov function that the quadratic equation (2.6 1 has exactly one strictly 
positive root. We denote this positive root as 2;+, so the computed gradient of the 
new optimal cost and the computed new optimal control at the new patch point are 



dx 



{x'+^) = z+n 



1 Stt^+i 



ix^+')gix^+') 



(2.7) 



2.6. Computing higher order partial derivatives of the optimal cost 
away from the origin. Let denote the computed optimal direction at the new 
patch point calculated from the first order partial derivatives of the computed optimal 
cost. If the exact and computed gradients of the optimal cost are sufficiently close 
at the new patch point, then it follows from the assumption that the optimal cost 
is a strict Lyapunov function that the computed optimal direction is nonzero. The 
computed optimal direction is calculated from the formula 



= f{x'+^) + g{x'+^)K'+\x'+^). 



We set 



f = [t>i . . . f"] 



where V^,V^,. .. , T^" form an orthonormal basis of M". This can be done with a 
Householder reflector jS] ch. 5.1.2]. We will compute the higher order partial deriva- 
tives of the optimal cost under the change of variables 



and then recover the partial derivatives with respect to the original state space vari- 
ables. In the new variables, the HJB equations (1.3) become 

dn 







where 



n{x'+^ + Vi) 

V^fix^+' + vi) 



m EE v^gix^+' + vi) 



(2.8a) 
(2.8b) 

(2.9) 



To derive a formula for the characteristic second order partial derivatives of the opti- 
mal cost, we evaluate the derivative of the flrst HJB equation in (2.8a) with respect 
to at ^ 







= 0, and arrive at 



(0)- 



(0)(^(0)- 
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(0)«:(0) 



^(0) + 2^(0)(«.(0))^ (2.10) 



where x denotes the optmial direction at x^^^ 
tive of K with respect to drop out due to (2.8b) 



The terms involving the partial deriva- 
We compute the characteristic 



second order partial derivatives of 7r'+^ by substituting in the computed approxima- 



tions to the optimal direction and gradient of the optimal cost into (2.10), and solving 



for the unknown second order partial derivatives of the optimal cost. This yields the 
formula 



(0) = -i 



1 



^(0)^^+1(0) 
1 df 



dq 
56 



(0) 



(0)(«'+i(0))' 



(2.11) 



The characteristic second order partial derivatives are a strict subset of all the sec- 
ond order partial derivatives, since the HJB equations do not specify all the second 
order partial derivatives. We call the remaining second order partial derivatives non 
characteristic. We compute them by inheritance, meaning for 2 < ji < j2 < n we set 
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^(0) = 



Vix'^' +V0]^^,- (2.12) 
To get the approximate first order partial derivatives of the optimal control, we follow 



the same process. We differentiate the second HJB equation(2.8b) with respect to 
at ^ = 0, and after substituting in the computed partial derivatives of the optimal 
control and cost, we obtain the formula 



dk'+^ , , 1 



-mm 



dn 



-(0)^(0)^^+1(0) 



df 



(2.13) 

We compute the remaining higher order partial derivatives in an analogous fashion. 



Their formulas are in J 9.2 



We now introduce new notation to describe how to recover the partial derivatives 
with respect to the original state space variables from the partial derivatives with 
respect to the new variables. They are computed by 



{x 



d_ 



(2.14) 



dx 7 , • • ■ Qx n 



-ix 



d_ 



(0 



where Cj is the j column of the identity matrix. Each formula in (2.14) is a standard 
dot product, written as the product of a row and column vector. The symbol has 
two meanings in (2.14). It is the standard Kronecker product if it appears between 

"l[7r*+i(^)] is new notation for the Kronecker 



two matrices. The row vector 



9? 



9? 



derivative, which is a bookkeeping mechanism. In the first equation in (2.14), if the 
state space is two dimensional then the Kronecker derivative is defined as 



d_ 



d_ 



ii) 



5=0 



We give the full definition of the Kronecker derivative in the appendix. 

2.7. Computing the optimal cost and optimal control away from a 
patch point. To compute the optimal cost or optimal control at a point in the 
state space that is not a patch point, we need a means to determine which patch the 
point belongs to. In other words, we need to know which polynomials tt* and to 
use to calculate the optimal cost and optimal control at some x in the state space. 
There are two types of patches, typical patches, and a single Al'brekht patch. The 
boundary of the Al'brekht patch is a level curve of tt" that encloses the entire patch. 
The optimal cost and optimal control at a point inside the Al'brekht patch are com- 
puted from 7r° and A typical patch is associated with the patch point x'^~^^ ^ 0, 
and the optimal cost and optimal control at every point in the patch is computed 
from 7r*+^ and When the state space is two dimensional, a typical patch has 

four intersecting boundaries that enclose the patch. Two of the boundaries are level 
curves of the optimal cost, and the other two boundaries are lateral boundaries. The 
patch point lies on the optimal cost level curve of tt' that defines one of the 
patch boundaries. The other optimal cost level curve patch boundary is a level curve 
of 7r*+^. Each lateral boundary is a straight line segment that intersects the level 
curve of tt' at a point Xb, and is oriented so that it points in the opposite direction of 
the optimal direction /(xf,) + g{xb)K^{xb)- When the state space is two dimensional, 
the patch boundary point Xb is placed on a level curve of the optimal cost so that it 
is halfway between two adjacent patch points that lie on the same level curve. The 
optimal direction at Xb may be computed from the approximate optimal cost associ- 
ated with either adjacent patch point. Illustrations of the two types of patches are in 
Figure [2T| 













^^^^^^^^^ 
















(a) Al'brekht patch 




(b) Typical patch 



Fig. 2.1. Al'brekht patch and typical patch 



3. Analytical error bound for the patchy method. The difference between 
the exact and computed optimal cost grows as a power of the maximum consecutive 
patch point distance times a term that grows exponentially as the patchy method 
moves along a sequence of consecutive patch points. We first develop some concepts 
and lemmas that we will need before making the more rigorous statement of the error 
bound in Theorem 13.61 at the end of this section. 

3.1. Sequence of consecutive patch points. The error bound is predicated 
on the notion of a sequence of consecutive patch points. 

Definition 3.1. The patch points x^ and x'^'^^ are said to be consecutive if the 
computed optimal cost polynomial centered at 2:*+^ was computed from information 
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contained in the computed optimal cost polynomial centered at . The origin is always 
the first patch point in a sequence of consecutive patch points. A sequence of three 
consecutive patch points is illustrated in Figure \3.1\ 




Fig. 3.1. A sequence of three consecutive patch points 



3.2. Notation. The patchy algorithm takes a truncated series solution to the 
optimal cost centered at the previous patch point, and generates a new series solution 
to the optimal cost centered at the next patch point. Thus, it can be interpreted as 
a mapping from a set of polynomial coefficients associated with the previous patch 
point to a set of polynomial coefficients associated with the new patch point. These 
coefficients are approximate partial derivatives of the optimal cost evaluated at the 
appropriate patch point, so we introduce notation to reflect this. 

denotes the vector of computed ^/"^ partial derivatives of the optimal cost at 
a;'. For example, if the state space is two dimensional 



dxi y-^ ) dx^dx^ I dxi y-^ > 



C" is the vector of all partial derivatives of the computed optimal cost at up to 
order d + 1, that is 

c^^{ci C\ ■ ■■ 

We denote with 0^ the part of the patchy method that computes the j*^ order 
partial derivatives of the optimal cost at the current patch point from the partial 
derivatives of the optimal cost evaluated at the previous patch point, meaning 



Cj denotes the vector of exact j**^ partial derivatives of the optimal cost tt at x*, 
and C* has a definition that is analogous to C*. 

3.3. Local truncation vector. Definition 3.2. C* denotes the vector of 
exact partial derivatives of the optimal cost at , and C^^^ denotes the exact partial 
derivatives of order j of the optimal cost at x''^^ , where x^ and x"^^^ are consecutive 
patch points. The local truncation vector rj is 

which is the difference between the exact j*'' order partial derivatives of n at x^'^^ , 
and the computed partial derivatives of tt at a;*+^ that are generated by feeding (f)j the 
exact partial derivatives of of tt centered at . 

3.4. The Lipschitz condition. Definition 3.3. Suppose Clj and Cj are vec- 
tors of exact and approximate partial derivatives of order j of the optimal cost tt at 
x^ . The function <j) is said to satisfy the Lipschitz condition if there exists a maximum 
multiplier Mmax > and a maximum consecutive patch point distance h > 0, such 
that for all M < Mmax and all \\x'^~^^ ~ x^\\ < h, then 

WMC') - MClh < LM\\x'+^ - forO<k<d+l 

whenever 

II q - q||2 < M\\x'+^ - .^is^'"' forO<j<d+l 



The following lemma will help us establish that the patchy method satisfies the 
Lipschitz condition. 

Lemma 3.4. Polynomials satisfy the Lipschitz condition of Definition with 
Mmax < oo . We may write any degree d + 1 polynomial as 

d+l 

P{C,x) = Y,Ce-Mx)- 



where Cg is a coefficient vector as defined in {3.2 and mg{x) is a vector of degree 
monomials so that all the degree I terms of the polynomial are Cg ■ mg{x). Thus, if 



\Cg-Cg\U < M 



\d+2- 



Qkp 



dxj-^ dxj^ ■ ■ ■ dxjf. 



' , then there exists Lmh < oo such that 

{C,x)- /'^^^^ . {C,x) < L,„,M||x|lf 2-^- 



Proof. Since P{C,x) — P{C',x) — P{C — C,x), we may assume without a loss of 



generality that C ~ 0, and ||C^||2 < -^II^^H 



d+2- 



By Cauchy-Schwarz, 



Qkp 



d+l 



dxj-^ dxj^ ■ ■ ■ dxji 



-iC,x)\<Y,\\Ci 



d^mg 



dxj-^ dxj^ ■ ■ ■ dxj^ 



(x) 



Each entry in the k order partial derivative of mg is 0{\\x 



Cgh < M\\x 



^, SO each term in the sum is C'(||a::||''+^ 
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, and by assumption 



□ 



The next lemma establishes a bound on the difference between the Taylor poly- 
nomial coefficients of the optimal cost tt, and the coefficients of the computed optimal 

cost TT*. 

Lemma 3.5. Suppose there exists a maximum consecutive patch point distance 
h > such that (j) satisfies the Lipschitz condition with constants L and Mmax- Also, 
suppose there exists a local truncation error constant T such that for any patch points 
and x^'^^ in \ A within a radius of h of each other, 

||r;|| < Th'^+'^-^ 

where the approximating polynomials to the optimal cost are degree d + 1. Let (7* 
denote the vector of j*^ order partial derivatives of the optimal cost computed by the 
patchy method, where i is the sequence index for a sequence of consecutive patch points. 
Let C* denote the corresponding vector of exact partial derivatives. The growth in the 
difference between the true and computed j*'' partial derivatives of the optimal cost is 
bounded by 

\\Ci - CiW < ^^-^Th'^+^-^ forl<i<N 
where N is the smallest integer such that 

Proof. The proof proceeds by induction on the sequence index i, and is similar in 
spirit to the construction of an error bound between the true solution of an ODE and 
a solution computed by a one step method [Tli ch. 7.2]. We assume that all arithmetic 
is exact. It follows from the definition of the local truncation vector that the difference 
between the true and computed j*^ order partial derivatives is bounded by 

liq+'-q+'ll<ll0,(c^)-0,((?^)ll + l|r;ii 

At the origin, the patchy algorithm computes the exact partial derivatives of the opti- 
mal cost by Al'brekht's method, so Cj = for < j < d+1. Therefore the theorem 
holds in the base case of the induction proof. Now, suppose that at the i*'' patch point 
in the sequence of consecutive patch points, 

liq - qil < ^^Th''+^-^ and ^^T < M^,, 

forO<j<d+l. Then 

\\cj,^iC')-cb,{&)\\<L^^Th'+'-^ 

for < j < d+1 since (j) satisfies the Lipschitz condition. Therefore, at the next patch 
point in the sequence, the difference between the partial derivatives is bounded by 

||C!;+i - C'+^\\ < L^^-^Th''+^-^ +Th''+^-' = ^^^—lTh''+^-^ . 



Thus, the conclusion of the lemma holds. □ 
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3.5. Analytical bound on absolute error of the optimal cost. Theorem 
3.6. Suppose there exists a maximum consecutive patch point distance h > such 
that (f) satisfies the Lipschitz condition. Also, suppose there exists a local truncation 
error constant T such that for any consecutive patch points and x^'^^ in Xc \ A 
within a radius h of each other, ||t*|| < Th'^^'^^^ , where the degree of all the optimal 
cost approximating polynomials is d + 1. Finally, assume that the maximum patch 
diameter is proportional to the maximum consecutive patch point distance. Then 
there exist constants L and K such that for any x in the i*^ patch in a sequence, the 
error between the exact and computed optimal cost is bounded by 



Hx) - 7r\x)\ < k(^^j---^ + 



and the difference between corresponding k order partial derivatives is bounded by 



dxj^ ■ ■ ■ dxjf. 



(x) 



dxj-^ ■ ■ ■ dxjf. 



(x) 



< K 



1 



L - 1 



d+2-k 



Proof. The basic idea behind the proof is to write the true optimal cost as the sum 
of its Taylor polynomial and a remainder term, then apply Lemma \3.5\ to bound the 
difference between the coefficients of tt' and the coefficients of the Taylor polynomial. 
By Lemma \3.5\ there is some Lipschitz constant L such that the exact and computed 
k*^ order partial derivatives of the optimal cost at x^ are bounded by 



\\Cl~Cl\\<^^Th''+^-'' 

where T is the local truncation constant from the theorem statement. Let x be any 
point in the i^^ patch in a sequence of consecutive patches, and let D denote the 
maximum patch diameter for all the patches in the computational domain Xc. By 
Lemma\3.4\ there exists Link such that 



Qk 



[P{C^-C\x-x^)]^^^ 



< L 



1. 



inh 



1 



-TD 



d+2-k 



By assumption, the optimal cost is smooth on Xc, so it follows from the definition of 
the Taylor remainder constant Rt that 



Qk 



dxj 



■ dxj 



[r:{:,-x^)-P^C\x-x%^__ 



< RtD 



d+2-k 



Therefore 



dxj^ ■ ■ ■ dxjf. 



(x) 



Qk^i 



dxj^ ■ ■ ■ dxji^ 



(x) 



< 



Qk 



^ -\PiC'~C\x-x')] _ 



Qk 



dxj-^ ■ ■ ■ dxjf. 



[n{x-x')~P{C\x~x^)]^^^ 
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The error bound from the theorem statement follows by setting K = {D /h)'^^'^ max{TLinh, Rt)M 
Our assumption that the patch diameter D is proportional to the maximum consecu- 
tive patch point distance h guarantees that D/h is bounded from above by a constant. 
□ 

In sections 4][7 we list sufficient conditions that guarantee that the local trunca- 
tion error of the patchy algorithm has the desired order, and the patchy algorithm 
satisfies the Lipschitz condition, thus guaranteeing the error bound in the previous 
theorem. 

4. Local truncation error for first order partial derivatives of the opti- 
mal cost. We state Theorem 4.1 whose content implies the first order local truncation 
vector is bounded by ||r{|| < and defer its proof until after we have established 

some supporting lemmas. In this section, we always assume that the conditions on 
the optimal control problem in |2.1| hold. 

Theorem 4.1. Suppose tt solves the HJB equations (1.3 1 under the assumptions 
in { 2.1 Let n denote the Taylor polynomial of the optimal cost centered at the previous 
patch point, and let {x^~^^) denote the computed gradient of the optimal cost at 

x^'^^ , computed from the coefficients of it. There exists a ball centered at the origin in 
which the Al'brekht patch must lie, a maximum consecutive patch point distance, and 
a constant T such that 



dn 
dx 



(x 



i+1 



dx 



-(x 



< T 



dx 



(x 



dx 



(x 



(4.1) 



whenever the distance from x^^^ to the previous patch point is less than the maximum 
consecutive patch point distance . The maximum consecutive patch point distance hx 
must be less than the consecutive patch point distance that we derive in Lemma\4^ 



and small enough to ensure W'^ix^'^^ 

diT 



^(x 

ax ^ 



< M^, where 



Ma < 



inf 



Ma < 



dx 

inf- i^'^ 



(x) 

S^Px,\A\\fix)\\ 



Ma < 



inf 



Xc\A\dx 



Ma < - sup 



^^Px,\A\\9ix)\ 
1 / dw 



Xa\A 

\\f{x)r 



^(.)||(II/(^)IP + 2^| 



l5(^)ll 



Il3(^)ll 



inf 

X,\A 



diT 
\dx 



( 



{x)x 



The assumptions in \2.1 guarantee that each right hand side of the previous four 
inequalities is strictly positive. 

Corollary 4.2. There exists a maximum consecutive patch point distance hx 
and local truncation constant T such that for any two consecutive patch points x^ and 
x^~^^ in the computational domain within a distance of Ht of each other. 



dn 
dx 



{x 



dx 



-{x 



< T\\x' 



\d+l 



(4.2) 



where ^gj' {x^'^^) denotes the computed gradient of the optimal cost at x^^^ , computed 
from the coefficients of the Taylor polynomial centered at the previous patch point. 
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Equivalently 



Ti\\= ci+'-Mci 



< Til 



\d+l 



It is helpful to think of the quantities /„ and .g„ from (2.5), and the solution to 
quadratic equation z from (2.6) as functions of a nonzero vector w, so we define 

(4.3) 



9n{w) 



g„(w) = 
r(x*+i) ff„(u;)^0 



As a consequence of the assumption that |^(x)i < 0, /„(w) and gn{w) cannot 
both be zero away from the origin if w is sufficiently close to ^{x^'^^)- Therefore, 
z{'w) is well defined. The difi^erence between the true and computed first order partial 
derivatives of tt can be rewritten as 



dx 



ix^+') 



i+l 



dx 



-(x^+i) 



dx 



and the main inequality (4.1 1 from the statement of Theorem 4.1 will follow from the 
fact that 



^(f (^'+^)) 
1111(^^+^)11 



= l + 0{ 



diT 
dx 



[x 



dx 



(x'+i)||) 



(4.4) 



as long as the distance from x*^^ to the previous patch point is less than the maximum 
consecutive patch point distance. 



To prove (4.4), we define the helper function 



(4.5) 



if(x^+i)+i(f (x^+i)-ff(x^+i)^ 

so that we can appeal to the mean value theorem, and rewrite the left-hand side of 
(|44| as 1 
bounded by 

dR, 



^(^) for some < ^ < 1. If i? is differentiable, then its derivative is 



dt 



it) 



< 



dz fdn 



dS-d-x^^^'^'K-d-x^^^-lTx^^^ 



dir , 



f(x)+t(ff(x)-ff(x) 
(^)-ff(^) 



dx ' 



f(^)+i(ff(^)-if(^); 



(4.6) 
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Our assumption that the optimal cost is a strict Lyupanov function is the key fact 
that guarantees that there exists a maximum consecutive patch point distance such 



that ^{t) is well defined for < t < 1 and (4.4) holds. This is the main content of 



Lemmas R~3l 4.4 4~5l and 4.6 



The next lemma guarantees that the denominators in the bound on ^{t) (4.61 



are bounded away from zero. 

Lemma 4.3. Let n denote the Taylor polynomial centered at the previous patch 
point. Ifn has continuous first order partial derivatives on Xc and is a strict Lyapunov 
function on Xc, then there exists a maximum consecutive patch point distance h' and 
constant T' , such that 



fg(^)-ff(^) 

if(-)+<ff(^)-ff(-) 



< T' 



dx 



(x) 



dn 

dx 



(x) 



whenever < t < 1 and the distance from x to the previous patch point is less than 
h'. 

Proof. We .split the proof into two cases, depending on whether the previous 
patch point is the origin or not. In either case, we will use the fact that if < 
&)-Bi^)\\<\\lEi^)lthen 



dx 



f(^)-ff(^) 



if(^)+i(ff(^)-if(^) 



< 



if(^)ii 



for allO<t<l. 

If the previous patch point is the origin, then we need establish the existence of a 
deleted neighborhood of radius h' centered at the origin such that 



dn 
dx 



(x) 



diT 
dx 



{x) 



dn 

dx 



(x) 



> 



for all X in the neighborhood. ^ is nonzero away from the origin under the as- 
sumption that t: is a strict Lyapunov function on Xc, so we need to establish that 
the difference between the gradients of tt and tt goes to zero faster than the gradient 
of TT as X goes to zero. The assumption ( ^2.1\ Assumption that Q >~ guaran- 
tees that the leading order linear term of §f^(a;) is nonzero for all nonzero x. Thus 



dx 



(^)ll/llf?(^)ll 



as X 0, so there is a radius h' > such that 



dn 
dx 



(x) 



dn 
dx 



[x] 



< 



dn 
dx 



{x) for all X in < \\x\\ < h' . 



Since ^ and ^{x) are continuous and r < \\x\\ < h' is compact, then for any r in 
< r <V, 



sup 

r<||a:||</i' 



diT 
dx 



(x) 



< oo. 



Thus, the lemma holds if the previous patch point is the origin, and the distance from 
x^^^ to the origin is no greater than h' . 
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For this lemma to apply to the patchy method, the Al'brekht patch A must lie in 
the neighborhood of the origin of radius h' , and the neighborhood of radius r must lie 
completely inside the Al'brekht patch. 

We now consider the case where the previous patch point is not the origin, so 
its distance to the origin is at least r. \\ §f-(a;)|| is bounded away from zero on Xc \ A, 



id lim^ 



^(x) — §f^(a;)|| = 0, so there exists an h' such that 



sup 



dx 



ix) 



< oo 



and we have established the validity of the lemma when the previous patch point is not 
the origin. □ 

Our next task is to establish the existence of Ma so that z(|^(x)+A)/|j |^(a;)+A|| 
is uniformly bounded for all || Aj] < Ma and all x in X^. This is the content of lemma 
Ol 



Lemma 4.4. 



sup 



||A||<Ma 



|z(ff(x)+A)| 



if(x) + A|| 



for any nonzero Ma satisfying 



Ma < inf 



dn 
dx 



Ma < 
Ma < 



{x) 

inis\^{x)f{x)\ 



sup5||/(a;)|| 



(4.7) 



inUc\^{x)g(x)\ 



SUP5c||5(x)|| 



where 



S = {xeXc\A 



dn 

dx 



{x)gix] 



< 



inf r{x) inf —(x)x 



dx 



The assumption that n is a strict Lyupanov function and the definition of S 
implies that both sup5||/(a;)|j and sup^c ||.g(a;)|| are strictly greater than zero, so Ma 
is well defined. 



Proof. Let w — (a^) + A, if w ■ g{x) ^ 0, then by the formula for z{w) in (4.3) 
^(^) w ■ fix) + ^{w f{x)Y + 2i^^{wg{x)) 



{w ■ g{x)Y 



(4.8) 



Since (§^(a;) + A)g{x) appears in the denominator of (4.8), we split the proof into 
the two cases where x is in S or its complement. 

If X is in S, then by the definition of S and the assumption that n is a strict 
Lyupanov function 



diT 

dx 



1 



inf 



' X^\A 
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dn 
dx 



{x)x 



< 0. 



Therefore, ifM^ satisfies the second condition in (4.7), then there is a strictly negative 
uniform upper bound on {'^{x) + A)f{x) for all x in S and all ||A| 



< Ma- It 

follows that for all x in S , the square root term in (4.8) has the Taylor expansion with 
remainder 



'iw-f{x)y 



2^-^iwgix)r 
r[x) 



<w-f{x)){l 



q{x) g{x) Y 1 
r{x)\wf{x)) VTTI 



for some ^ > 0. So, for all x in S and all || A|| < M/\ 



^(f (x) + A)| 



q{x) 



\\^{x) + A)|i {^{x) + A)f(x) VT+I - ( 



q{x) 



A)/(a:) 



(4.9) 



which also holds in the case where {jj^(x)+A)g{x) 



0, and consequently z{^{x)+A) 



solves a linear equation ( |2.6[ ). The supremum of the right-hand side of (4.9) over S 
and all A < Ma is finite. The first two conditions on Ma (4.7) guarantee that the 
infimum of + A)/(a;)|| is strictly greater than zero over S and all A bounded 

above by A/a in norm. The function q is continuous on the compact set \A^S, 
therefore its supremum over S is bounded. 

We now turn to the case where x is in . In this case, the only way z{w)/\\w\\ 
(4.8) can become unbounded is if {§^{x) + A)g(x) can become arbitrarily small, but 



C 



IS 



this cannot happen. By the definition of S , the infimum of \^(x)g{x)\ over S' 
strictly positive, and the infimum of + A)(7(a;)| over and all A bounded 

above in norm by M\ is also strictly positive. 
□ 

Our final task in establisliing Theorem |4.1| is to show that there exists an Ma 



such that the gradient of z(w) exists and is uniformly bounded over ^^^(Xc \ .4) + A 



for II A|| < Ma. This is the content of Lemmas 4.5 and 4.6 . 

Lemma 4.5. Under the assumptions on the optimal control problem of ^2.1\ the 



function z that solve s (|2.6| i s continuously differentiable in a nonempty neighborhood 
of every point in ^ {Xc \ A) . 

Proof. By the implicit function theorem |3 ch. 9], a solution to ax^ + bx + c = is 
a continuously differentiable function of its coefficients in a neighborhood ofao, bo, and 
Co if xq solves the quadratic equation with coefficients qq, bo, andco, and 2aoXo + bo ^ 
0. To conclude that z{w) is continuously differentiable in a neighborhood of each point 



ax 



':{Xc \A), we must verify that 
for every x in ^ {X^ \ A) . 



1 

r{x) ' 



As a function ofx, q^x is continuous under the assumptions that f, g, r, and 

are continuous on Xc \ A. Furthermore, is nonzero on Xc \A by the assumption 
that t: is a strict Lyapunov function, so there is a nonzero uniform lower bound on 
both |^(a;)a;| and \\^\\ on Xc \ A. It then follows from the identity 



-^.g„(£(:.))^(£(x) 



that -^<?n(if (x))2z(ff (x)) + /„(ff (x)) IS nonzero for every x m ff (X, \ A). □ 
We now verify that there exists an A/a > such that 
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fe(ff(^c\^)+A)|| is 



uniformly bounded for all ||A|| < Ma- 



Lemma 4.6. If M/^ satisfies the conditions of Lemma 4-4 (4.7| in addition to 
dn 



Ma < I sup ( 



dx 



(x) 



(11/(^)11 



r{x) 



i3(^)ir+ii/(^)ir + 113(^)11 



X inf {^{x)±] 



then 



sup 
||A||<Ma 



dz ( dn ^ 
dw \ dx 



x) + A 



< oo. 



Proof. Let w = ^(x) + A. By Lemma 4-5. §^ exists in a neighborhood of j^{x 



for all X in \ A, and an implicit formula for it is 



dz 
dw 



(w) 



fn{w) - :^gn{w)'^z{w) 



(iTTr) 9niw){\\w\\2g{x) ~ gn{w)i 



z{w) 



\W\\2 



fi^) 



(4.10) 



Both f and g are uniformly bounded on Xc \ A. Since A < A/a < inf ^ ^^ || ^(x)|| 



w is nonzero, and the terms fn{w) and gn{w) are both uniformly boun ded o n Xc \ A 
for all A < Ma by the supremums of f and g on Xc \ A. By Lemma 4-4 the term 
z{w)/\\w\\ is bounded. 

The only remaining term that can potentially become unbounded is (fniw) — 



:p^9n{w)'^z{w)) ^. This quantity can be rewritten as 



( f I N 9n(wY Y r t .2 , r,<l[X) , . 



q(x)_ 



/dn 

\\^{x) + AW^\\dx 



{x)xy +E{x,A)'^ 



(4.11) 
(4.12) 



Equation (4.11) follows from the fact that both sides of the equation are formulas 



for the discriminant of the quadratic equation (2.6). // we solve the HJB equation 
(1.3a) for q{x) in terms of ^{x) and the other problem data, and then substitute this 



formula into ( |4.11 1, then we arrive at (4.12). We defined Ma in the lemma statement 
so that E[x, A) = OdjAH) and the infimum of (4.12) is strictly greater than zero over 
Xc\A and all ||A|| < Ma- □ 

5. 01 satisfies the Lipschitz condition. Theorem 5.1. Under the assump- 
tions on the optimal control problem of ^2.1\ there exists a maximum step size hj^ > 0, 
maximum multiplier Mmax > 0, and Lipschitz constant Li such that for all x^ and 
x^~^^ in Xc \ A within a distance of h^ of each other, and for all M < Mmax, 



\MCi - MC'IW < L,M\\x 



i+1 



\d+l 
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whenever 



liq-qil <M\\x'+'~x'\ 



forO<j<d+l 



where C* is the vector of exact partial derivatives up to order d+l of the optimal cost 
at x^ . The theorem holds for any Mj^ax, pair that satisfy maximum consecutive 
patch point distance multiplier condition of Definition \5.S\ 

Definition 5.2. The maximum consecutive patch point distance hj^ and the 
maximum multiplier Mmax o-fe said to satisfy the maximum consecutive patch point 
distance multiplier condition if 



hr > 



Mr, 



> hL<hT {V^L,nhM,nax + T)h'^+^ < Ma 



where Link is defined in Lemma \3.J\ hT and T are the maximum consecutive step size 
and local truncation constant from Corollary \4-S\ and satisfies the inequalities of 
Theorem \4-l\ 

Proof. [Proof of Theorem 5.1 If w ^{C, a;*+^ — x*) is nonzero and z is defined 



at w, then 0i can be expressed as 



MC)^zi^ — {C,x^+'-x^))- 



dx 



{C,x'+^-x') 



dP 

dx 



(C, x'+i - x^ 



where P is the polynomial whose coefficients are held in C , and is defined in Lemma 
3.4 We will show that there exists L' such that, if the conditions of the theorem are 
met, then 



dx 



{C\x' 



x') 



yi{c\x^ 



\\^{C\x^+^-x^)\\ \\jP[C\x^+^-x^)\\ 



< L[M\\x 



i+l 



(5.1) 



and 



:(^dP{C\x'+' - x^)) - z(^^{C\x'+' - x'))\< L[M\\x^+' - xY+' (5.2) 
which implies the conclusion of the theorem. 



We start with the inequality in (5.1). P and all its partial derivatives are linear 
in its first argument. If 



f)P BP ~ 

^iC\x'+' ~ x') - "^iC\x'+^ - x') 



< 



dP 
dx 



{C\x'+^ ~x') 



then 



9P^C\x'+^ ~x') 



r^{C\x^+^~x^)\\ |||f(C^x»+l-.T')| 

< 2 



|||f(C^x^+l~x')||-|||f(C^-(7^a:*+l-a;»)|| 



< 2- 
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where the final inequahty foUows from Lemma 3.4 The inequality in (5.1 1 will follow 
after we establish that for some e, 



< e < 



9P, 
dx 



dP 
dx 



-C\x'+' -x') 



(5.3) 



(5.3) also guarantees that ^(C*, a;*"*"^ — x*) is nonzero. C* holds the exact partial 
derivatives of the optimal cost at so P(C ", a: — a;*) = Tt{x) where tt is the Taylor 
polynomial centered at x*. By Corollary 



4.2 



dx 



9P, 
dx 



{C\x'+' ~x') 



< 



< T\W+' -X 



By Lemma [3^ 



(^P /9P 



Inequality |5.3| follows from the final strict inequality in 



dP_ 
dx 



{C\x 



dP_ 
dx 



dn 
dx 

dn 



> inf 

> 



(a;) 



9a: 

97r 
dx 



dP 

dx 

dP 
dx 



{C\x'+'^ -x') 



The maximum patch point multiplier condition (Definition 5.2) was defined so the 
last inequality is strict. 

We now prove that the inequality (5.2) holds by applying the mean value theorem 
to z{w). If 



f(c-,.-«-.-)-|,..«) 



< Ma 



(5.4) 



and 



< Ma 



(5.5) 



then for all r in < r < 1, 
dP , 



(l-r)-(CN. 



T^{C\x'+^-x')-^{x'+^) 



dx 



dx 



< Ma 
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meaning that every point on the line segment connecting ^(C*,x*^"'^ — a;*) and 



dP 



{C\x 



is at most a distance of Ma from Therefore, z is well de 



fined at the endpoints, and by Lem ma|4.6| there is a uniform bound on the magnitude 
of -i^ over all such lines. Inequality 5.2| will follow once we show that the conditions 



dw 



of the theorem guarantee that inequalities 5.4 and 5.5 hold. By assumption, < /it, 
so by Corollary |4.2| and Definition |5.2| 



and 



dx 



dP 

dx 



x^) 



< Th'i+^ < Ma 



dx 



{x 



dP 
dx 



< 



{C\x 



9P 

dx 



C\x'+^ 



< Ma- 



x^) 

d+1 



dn 

dx 



ix 



dP , 
dx 



{C\x'+' -x') 





6. Local truncation error for second and higher order partial deriva- 
tives . Theorem 6.1. Let C and C*^^ denote vectors of exact partial derivatives 
of the optimal cost at the consecutive patch points x^ and x^~^^ in the computational 
domain X^- If the optimal control problem satisfies the assumptions of ^2.1\ , then 
there exists local truncation constant T2 < od and maximum consecutive patch point 
distance h > such that for all 2 < j < d + I, 



\c: 



j\\d+2-j 



whenever \\x 



i+l 



x'W < h 



We delay the proof of Theorem 6.1 until after we have developed an overview of its 
main ideas. 

Let x'''^^ denote the computed optimal direction at the current patch point x = 
a;'"'"^. There are two changes of variables associated with the computed and exact 
solutions at the current patch point. They are 



x = x'+^ 



X 

W\ 



(6.1) 



We denote the computed and exact optimal cost and control under the changes of 
variables as 



-i+l 



TT 



[X^ 



n{0 = tt{x'+^ + V£) 
k{0 = n{x'+^ + VO- 



(6.2) 



We defined the Kronecker derivative notation ^ 

ox 



^ in ^ 

OX 



2.6 



and the 

appendix, and it is merely a bookkeeping mechanism for mixed partial derivatives. 
The patchy algorithm computes the fc"^ order partial derivatives with respect to the 
change of variables of the optimal cost, then recovers the partial derivatives with 
respect to the original state space variables by the formula 



d_ 
dx 



k til 



A, 

dx^ 



d_ 

di 



k times 



(6.3) 



k times 
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The exact fc"^ order partial derivatives of the optimal cost with respect to the original 
state space variables are recovered by 



k times 



k times 



Oi^ k times 



(6.4) 



There are two main ideas behind the proof of Theorem |6.1[ The left hand side 



of (|6.4|) is the same for any choice of orthogonal matrix V in the change of variables 
VS^. In the case of the computed partial derivatives of the optimal cost, 



X = X 

the entries in 



^ (81 ■ ■ • €5 J| appearing on the right hand side of (6.3 ) are a mixture of 
characteristic and non characteristic partial derivatives. Despite this, the computed 
partial derivatives with respect to the original state space variables of the optimal 
cost are invariant under any choice of orthonormal vectors in the change of 

variables x = a;'+^ + as long as = This is the first main idea and 



is the content of Lemma |6.2[ Corollary |6.3[ and Lemma |6.4| 

The second idea is that there exist two convenient changes of variables that sim- 
plify our task of establishing the desired bound on the difference between the partial 
derivatives with respect to the original state space variables of the exact and computed 
optimal cost by bounding the difference between the partial derivatives under the con- 
venient changes of variables. The advantage of this approach is that we may treat 
the case of bounding the difference between the computed and exact characteristic 
partial derivatives independently of the non characteristic partial derivatives. 

Once we have established these two facts, we will prove the theorem for the second 
order partial derivatives of the optimal cost and proceed by induction. 



The relevant interpretation of Lemma 6.2 is that, taken as a group, the charac- 
teristic partial derivatives that the patchy algorithm computes at the current patch 
point can be expressed as an orthogonal transformation of a vector that depends only 
on partial derivatives of the problem data at the current patch point, and on partial 
derivatives of the optimal cost that are of strictly lower order than the characteristic 
partial derivatives being computed. Secondly, all these partial derivatives are with 
respect to the original state space variables. 

Lemma 6.2. Suppose V is an orthogonal matrix whose first column is the exact 
normalized optimal direction, — x/\\x\\ at x = x'""^^ . Let 7r(^) denote the optimal 
cost under the change of variables x 



vectors G M"'' independent of V'^ , . . . , V" 

partial derivatives of the optimal cost satisfy 



Then, at x — x^^^ , there exist row 
, such that the exact characteristic 



d 



dw 



d d [ dn 

— ® — 



(0 
(0 



4=0 



- {w^ix'+')fv 



{w^x'+^)fv (E}V 



(6.5) 



d 



dn 



id ^{wP{x^+')fVi 



where w'^ix^^^) depends only on partial derivatives of the problem data and partial 
derivatives with respect to the original state space variables of the optimal cost of order 
k — 1 and lower. 



21 



An analogous statement is true for the computed characteristic partial derivatives 
of the optimal cost. 

Proof. To reduce the number of terms in some of the ensuing formulas, let F and 
i denote the dynamics and lagrangian as functions of the new coordinates. 



Fio^fio+mm) 



where the tilde functions appearing on the right hand sides are defined by the change 
of variables x — x^^^ + V^. 

We will prove the lemma in the representative case of the third order characteristic 



partial derivatives of the optimal cost. It follows from the HJB equations (2.8a) that a 



formula for the exact third order characteristic partial derivatives of the optimal cost 



(0) = ± 



d'^TT dP a^vf dF 

;^(0)^(0) + ^^(0)^(0) 

dn d^F 
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(0) 



(6.6) 



The right hand side of the formula (6.6) contains three distinctive types of terms 
characterized by the order of the partial derivatives of tt, F, and £. The first type is 
the product of a second order partial derivative of tt and a first order partial derivative 
of F, the second type is the product of a first order partial of tt and a second order 
partial of F , and the final type is a second order partial of I. 

We will show that each type of term can be written as the product of a row vector 
that depends only on the partial derivatives of both tt^x) and F(x) at x — a;*+^ and 
the column vector V^^ V^^ . We tackle the first type of term first. A formula for 
a0,(O)||(O) ^s 



d^TT dF 



^J2 



n 



^ \dx 

,(7 — 1 



dx„ 



(x) 



(- 

\dx 



F.ix) 



(6.7) 

The quantity on the right hand side of (6.7) depends only on second order and lower 
partial derivatives of Tr{x) and F{x) evaluated at To derive a formula for 

(0)^(0) with the desired factor V^^igiV^^ , we use the fact that for every permu- 

^02 Jl 

tation (11,12) o/(ji,j2), there exists a permutation matrix P, that depends only on the 
permutation and not the choice of basis vectors, such that V''^ (Ei V^^ — P{V^^ <Si V^^). 
Therefore, a relevant formula is 



dF 
(0)7^(0) 



n 

y(- 



dn 
dx„ 



ix) 



(- 
\dx 



Fa{x) 



P' 



V'^ (g) 



Although tedious, we can derive formulas for the other terms in (6.6) that are 
the product of a row vector that depends only on the second order and lower partial 
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derivatives of the optimal cost and problem data with respect to the original state space 
variables, and the column vector V^'^ and V^^ . It follows that there exists w'^{x^^^) G 
such that 

^'"^ -(0) = {w^ix'+^))^V V^' 



9? 



in the theorem statement follows. 



so the formula for ^ 

The proof for the computed characteristic partial derivatives is analogous. □ 
The relevant interpretation of Corollary |6.3| is that, under two changes of variables 
that share the normalized computed optimal direction at the current patch point as 
their first basis vector, the two sets of computed characteristic partial derivatives of 
the optimal cost at the current patch point are related to each other by an orthogonal 
transformation. 

Corollary 6.3. If V and V are are orthogonal matrices whose first columns 
are the normalized computed optimal direction at x — x that is ^1 = ^1 = 
a;''*""'^/||i*^"'^||, then the approximate second and higher order approximate characteristic 
partial derivatives of the optimal cost computed under the changes of variables 



are related by the orthogonal transformation 



d_ 
dk 



= (g) • • • (g) 



d 



d_ 



d 



(0 



4=0 



Proof. The proof for the case of the approximate third order partials of the optimal 
cost is representative. By Lemma 6.2, there exists a single vector w'^'^^ that depends 
only on the previously computed lower order partial derivatives of tt^~^^(x) at x — 
such that the characteristic partials under both coordinate transformations are given 
by 



d d 



(0 



T ~ 



V 



d 



d 



(0 



V ®V. 



di di. 

The conclusion of the corollary immediately follows from the identity 



V) {V^V ( 



□ 

The content of Lemma 6.4 is that, under two orthogonal changes of variables that 
share the normalized computed optimal direction at the current patch point as their 
first basis vector, the two sets of computed characteristic and non characteristic partial 
derivatives are related by an orthogonal transformation. Thus, it follows immediately 



from (6.31 that the computed partial derivatives of the optimal cost with respect to 



the original state space variables at the current patch point are the same under two 
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different orthogonal changes of variables as long as the changes of variables share the 
normalized computed optimal direction as their first basis vector. 

Lemma 6.4. If V and V are orthogonal matrices whose first columns are the 
normalized optimal direction at x — x^^^ , that is = = x'''^^ /\\x'^~^^\\, then the 
approximate second and higher order approximate partial derivatives of the optimal 
cost computed under the changes of variables 

X = + y| X = x'+^ + 

are related to each other by 



Proof. The conclusion of the theorem will follow once we establish that the quantity 

- ^ ® • • • ® ^ [^'^Hi)]i^oi^^V ® • • • ® V^V) (6.8) 

is zero. We will prove the lemma for the third order partial derivatives of the computed 
optimal cost, which is a representative case. 

It follows immediately from the definition of the €5 • • • €D gj: notation that each 
entry in the term appearing on the left of (6.8) can be extracted by multiplying by one 
of the standard basis vectors. Any standard basis vector in M" can be expressed as a 
Kronecker product of three standard basis vectors in M", so 



g3~i+i 8 8 f) 

(0) = ^ « ^ ® :J[-'+^(0].=o^.. 



d^nd^A 9^ 9^ 

where Cj denotes the column of the identity matrix. We will establish that the 
difference in (6.8) is zero by multiplying it by Cj^ ® Cj^ <Si Cj^ and showing that 

is zero for any 1 < ji,j2ii3 ^ which is equivalent to showing that 

j>'*r\ (0) 

- E E E { g, (0)(^" • ■ ^'^)(^" • ^'')) (6-9) 

0-1=1 0-2 = 1 0-3 = 1 ?Ojj SOjj SOj3 

is zero. 

The partial derivatives 0/ 7r*+-'^(^) and 7r*+-'^(^) at = ^ = are computed by two 
different methods depending on whether they are characteristic or non characteristic 

24 



partial derivatives, so the proof that the quantity in (6.9) is zero naturally breaks up 



into two cases. In the characteristic case, at least one of the coordinate indices ji, j2, 
or j3 is one, and we may assume that = 1 without a meaningful loss of generality. 
The columns of V and V are both orthonormal sets, and = by assumption, 
so V"'^ ■ = '^1,0-3; where Si^a-^ is the Kronecker delta. Therefore for any ji,j2 in 



1 ^ ji,j2 < n, the difference in (6.9) reduces to a quantity that depends only on 



characteristic partial derivatives of the optimal cost 



n n 



+1 



— (0) 



9^1 J«=o^ ' (6.10) 



= 0. 



The first equality follows from the definition of the 



dx 



equality follows from Corollary Therefore the quantity in (|6.9|) is zero whenever 



S- , and the second 



a^. 05. 9{. 



(0) is a characteristic partial derivative. 



We now show that the difference in (6.9) is zero in the non characteristic partial 



derivative case, that is, when none of the coordinate indices ji, j2, or are one. By 
a standard argument 



d d 



d 



dx dx dx 



The right hand side of the previous equation is the product of a row and column vector. 
Since none of ji , j'2 , or jj, is one, the previous equation can be rewritten as 
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n n n ^ 

i: i: E(i£«|:<[''w].„..«(*'"'«'>"«<™)) 

(Ti— 2 o"2— 2 173 — 2 



d 



X (^{v"' ■ • v^^)(v''^ ■ (6.11) 

where the summations all start at two because ■ V^^ = ■ V^'^ = ■ V^^ = 0. 



Thus, the sum on the right hand side of (6.11 1 depends only on non characteristic 

(0) can 



partial derivatives o/7r*^^(^) at£,—0. Therefore when ji, j2, js 7^ 1, 

be expressed as a linear transformation 0/ (8" • • • (8) '^['^^^^{0]^=o! whose entries 
are composed of both characteristic and non characteristic partial derivatives. The 



conclusion of the theorem follows from the fact that (6.IOI and (6.11 1 cover all of 



( |6.9[ ). □ 

We now turn to the task of constructing two convenient changes of variables, 



meaning we can pick V and V in (6.1 1 such that V and V are almost inverses of 
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each other. This is the content of Lenima |6.5[ which in turn rehes on Lemma |6.6[ and 
Corollaries 16.71 and 16.81 

Lemma 6.5. Let denote the optimal direction at x = x''^^ computed by the 
patchy algorithm from the exact partial derivatives of the optimal cost at the previous 
patch point. If the optimal cost t:{x) is a strict Lyapunov function and the distance 
between the current and previous patch point is less than the maximum consecutive 
patch point distance h > 0, then there exists a constant B < oo and orthogonal 
matrices V and V such that 



X 

WW 



and 



\\V^V-I\\ < B\\x'+^ - x'\\'^+^ 

where the degree of the approximate optimal cost polynomial 7r*+^ is d + 1. 

Proof. By Theorem \4.1\ there is a constant T < oo and a maximum consecutive 
patch point distance h > such that 



dx^ ' dx ^ ' 



< T\\x 



Therefore, there exists an e > such that, for any x^^^ in \ A within a radius of 
h of the previous patch point 



< e < ILill -\\x-x 



i+1 1 



The result o f th e le mma follows from Lemma 6.6 where V and V are constructed, and 
Corollaries 



6. ? anc 



6.. 



where the bound on \\V V — I\\ in the lemma statement is 
established. □ 

We construct orthogonal matrices V and V associated with the changes of vari- 
ables for the exact and computed optimal cost such that and V are almost inverses 



of each other. This is the content of Lemma 6.6 which is mostly a restatement of [TUl 
pg. 73] in terms relevant to our problem. 

Lemma 6.6. Suppose v and v are nonzero vectors in M" such that v ■ v > 0, then 
there exist orthonormal bases {V^, . . . , V"} and {V^, . . . , V"} with the properties 
1. = and = vlWvW 



2. 
3. 



and V — ij/ 

The sets {V^ , . . . , y"} and {V^ , . . . , V^"} satisfy the orthogonality condition 
(yj . yf') = whenever j and -V^ < V ■ < 1 for 2 < j < n 
Proof. The first property from the lemma .statement is trivially achieved by setting 
and to the unit vectors in the direction of v and v. 

Let W be any n x n — 1 matrix whose columns are orthogonal to , and define 
W analogously. Let M and AI be the orthogonal matrices 



M 



- Ft/I 



w] 



M 



w] 



then lif^ M is orthogonal. The second statement of the lemma is just a restatement 
of the fact that the first row and column of M both have unit two-norm. 
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The third statement in the lemma statement requires a special choice of bases for 
the suhspaces orthogonal to and . Let W^W have the singular value decompo- 
sition 



where is ann—lxn—l diagonal matrix. If we take the sets {V^}"^2 '^"'^ {^"'}j=2 
to be the columns ofWU andWU, then {V^}"^2 '^'^'^ {^"'}j=2 '^^^ orthonormal bases 
for the span of the spaces orthogonal to and . Furthermore, it follows from the 
fact that {WU)'^WU = E that {1^^};=2 and {V^}^^2 are biorthogonal and ■ is 
a singular value ofW^W when 2 < j < n. 

Finally, we must verify that ■ < ■ < 1 for 2 < j < n. Let N and N 
be the orthogonal matrices 



N = [V^ WU] 
Their product has the form 



N EE [X>i WU] 



Each column of of N'^ N has unit norm, in particular the last column contains crmin{W^\ 
the smallest singular value ofW^^W, and thus satisfies 

1 < armniW^W)^ + \\iVYWU\\^ 



mm 

j 



It then follows from the final inequality and the assumption that (V^ ■ V^) > that 
yj . VI > . \/i for 2 < i < n. □ 

establish a bound on llV^-^y — /|| in terms of the difference 



Coroharies 



6.7 



and 



6.8 



between the true and computed optimal directions at the current patch point. 

Corollary 6.7. Let V and V denote the orthogonal matrices whose columns 



are {V^}^^i and {V^}"^^ from Lemma 



6.6: then 



\I -V^V\\2<2V2Jl-Vi-Vi 



Proof. Partition the matrices V and V as 

V = [Vi w] 



V = [Vi w] 



W and W were constructed so that their columns have unit norm and satisfy ■ 
= whenever j ^ k. Let S — W'^W , where S is diagonal and its diagonal entries 
are Wj ■ Wj . I — VV can be expressed as 



1 - Vi • Vi 
/ - E 
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V^W 
W^Vi 



(6.12) 



By Lemma 6. 6 . = Wj ■ Wj > Vi-Vi, so the norm of the diagonal matrix appearing 
in the right hand side of (6.12 I is bounded by 



1-^1-^1 
/- E 



<l~Vi-Vi 



(6.13) 



Let X be a unit column vector partitioned as x = [xi X2]'^ , then the following bound 
on the norm of the second matrix in (6.121 follows from the fact that ||V\"^W^|p = 
\\W^V,\\^ = l-Vi-V^ 



v^w' 

W^Vi 



= \\Vi^Wx2\\^ + \\W^ViXi\\^ 

< \\V,^W\\'\\x2f + \\W^Vifil^\\x2f) 

< 2(1 - ■ Vi) 



(6.14) 



The conclusion of the lemma follows from ( 6.13^ and (6.14). □ 

Corollary 6.8. Suppose v is a nonzero vector, and — 'L'|j2 < \\v\\2, 
exist orthogonal matrices V and V such that their first columns are 



then there 



V = 



and 



\I-V^V\\ < 4, 



Proof. The corollary follows from Corollary |6.7| and the fact that 



< 2, 



v\\ 



whenever \\v — v\\ < \\v\\ 





The relevant interpretation of Lemma 6.9 for the purpose of proving Theorem 6.1 
is that V^V and the repeated Kronecker product V^V of the orthogonal matrices 
from the convenient changes of coordinates are both perturbations of the identity 
of the same order. This fact is useful for bounding the difference between partial 
derivatives of the same function under the two changes of coordinates. 

Lemma 6.9. If A and B are matrices such that AB is mM"^" and \\AB — In\\2 < 
1, then 



\{ABi 



AS) -/„.||2 < {2'' - l)\\AB - IJ2 



where the Kronecker product on the left hand side of the inequality is repeated k times, 
and I„ and I^k denote the identity matrices in M"^" and M" ^" . 

Proof. The proof of the lemma rests on the fact that for any two matrices C and 
D, \\C®D\\2 — ||C'||2||-D||2- A standard induction argument establishes the conclusion 
of the lemma. □ 

We now turn back to the task of bounding the difference between an exact and 
computed partial derivative of the optimal cost with respect to the original state space 

28 



variables at the current patch point. This difference is given by the formula 



d d d 



d_ 



d_ 



V^V ~I) 



V'^e.j,®---®V'^ej^. (6.15) 



There are a few things to note about the right hand side of (6.15). The first is that 



it is the product of a row vector on the left and column vector on the right, where 



the column vector V'-^e,- 



V eo,, has unit norm since it is the column of an 



orthogonal matrix. It follows from the Cauchy-Schwarz inequality that the the right 



hand side of ( 6.15 ) is bounded by the norm of the row vector. This row vector has two 



terms, the first is the difference between the exact and computed partial derivatives 
of the optimal cost with respect to the changes of variables. The second is involves a 
vector composed of partial derivatives of the exact solution times a small perturbation 
of the identity. It follows from assumption that the optimal cost is smooth on the 



compact computational domain and by Lemma 6.5 that there exist T' < oo such that, 
whenever the current patch point is within a radius of the maximum consecutive patch 
point distance of the previous patch point, this term is bounded by 



d_ 



< (2' 



< T'Wx 



V'^V - I) 



l) sup 



d_ 
dx 



dx 



tt{x)] \\V^V-I\\ 



(6.16) 



This bound is independent of p, the order of the partial derivative, when p > 2. The 
remaining work is in establishing the existence of some T" < oo, such that under the 
convenient changes of variables 



d_ 



d_ 



<9e 



< T" 



(6.17) 



whenever the distance between the current and previous patch point is less than the 
maximum consecutive patch point distance. We will bound the left hand side of 



(6.17) by bounding each of its entries by considering two cases, depending on whether 



the entry is the difference of exact and computed characteristic or non characteristic 
partial derivatives of the optimal cost. 

With this approach in mind, we will need a bound on the partial derivatives of 
the problem data with respect to two changes of variables. This is the content of 
Lemma 16.101 

Lemma 6.10. Suppose all partial derivatives of s : M" up to order k exist 

and are continuous in a neighborhood of x in M" . // V and V are orthogonal matrices 
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and s and s are defined as 

Sii) = s{x + V^) 

then 



s{0 = -<^ + VO 



-(0) 



< 



(2'^ - 1] 



d_ 
dx 



If the components of G : M" — >■ M" all have partial derivatives up to order p in a 
neighborhood of x in M" , and G and G are defined as 



G(0 = V^G{x + VO 



G{0 = V'G{x + VO 



the 



J'^^ (0)- (0) 



< 



max 



d_ 
dx 



^[Giix)l=s \\V^V-Ih. 

ox 2 



Proof. A k order partial derivative of s can be expressed in Kronecker notation 



dx 



[s{x)]^=s{V(E)---(S)V){ej, (»' 



so after a sequence of algebraic manipulations, the difference between the partial 
derivatives of s and s is 



d'^s 



d d 



(0) 



dx 

< (2^ - ll 



dx 



d_ 
dx 



dx 



[s(^X^^x—x 



\VV^ - III 



The last inequality follows from the Cauchy-Schwarz inequality, Lemma \6.S\ and the 
facts that V'^ ■ ■ ■ ® V^" has unit norm and \\VV'^ - I\\2 = \\V'^V - I\\2. 

The difference between the partial derivatives of G under the two changes of vari- 
ables is bounded by 



Qk 



Qk 



^ ' e dx 
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d 



dx 



[G,{x)h=s \\VV^-I\\2. 



This bound is derived by using the result of the first half of the lemma to bound the 
infinity norm of the difference between the partial derivatives. The second result of 
the lemma follows from the previous bound and the identity 

G(0 - G(|) = [G{x + y|) - G{x + Vi) + {VV^ - r)G{x + VS,)] . 

□ 

The next corollary differs from the previous lemma in that it bounds the difference 
in partial derivatives of two different functions with respect to two changes of variables. 

Corollary 6.11. Suppose all partial derivatives of s,s : M" R up to order 
k exist and are continuous in a neighborhood of x in M" . // V and V are orthogonal 
matrices and s and s are defined as 



S(0 = six + 



siO = s(x + VO 



the 



□ 



-(0) 



d 



-(0) 



dx 



< 



dx 



[s{x) - s{x)]x^s: 



+ (2^= - 1) 



d_ 
dx 



dx^ 



Proof, 
d'^s 



d'^s 











( ^ ® 


d r 




\dx 


■■^dx^' 




d 


d 




dx 


•••«) — 

dx 


< 


d 


d r 


— (g) • 
dx 


■■^dx^' 




+ (2*= - 


dx ^ 



[s{x)]^=s{^ 



V^V 



(g) • • • (g) V^v) {cj^ 



dx 



^{x')\x—x 



We now have all the ingredients necessary to prove Theorem 6.1 



Proof. [Proof of Theorem 6.1 We denote the problem data under the two changes 
of variables in (6.1) as 





V^f{x'+^ + vo 


hi) ^ 


V^fix 


'+1 + v^) 




V^g{x'+^ + V^) 


m ^ 




+1 + vi) 


m ^ 


q{x'+^ + VO 


m ^ 


q{x'+^ 


+ vi) 




r{x'+^ + V^) 


Hi) ^ 


f{x'+^ 


+ vi) 



(6.18) 



We first consider the case of the second order partial derivatives of the optimal cost, so 
we must verify that the left hand side of (6.17) is C'(||a;*+-'^ ~x'^\\'^). The characteristic 
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second order partial derivatives of the optimal cost are given by (2.11) and the com- 
puted optimal control is given by (2.13). The exact second order partial derivatives 
of the optimal const and the optimal control satisfy the equations 



— (0) = -^ 



(o)(^(o) + |^mo) 



di 



1 dr 



(O)S(O)' 



and 



^(0) 



1 



'dn 



(6.19) 



(6.20) 



The difference between §|^(0) and gf:^(0) can be bounded in terms of two types 

of differences. The first type involves terms that are the difference between the partial 
derivatives with respect to the two changes of variables of the problem data. 



dl 
dq 



(0) 
(0) 



5/ 



(0) 



df 



(0) 
(0) 



dg 



df 

di 



j(0) 



(6.21) 



(0) 



If the distance between the current and previous patch point is less than the maximum 
consecutive patch point distance h, then by Lemmas |6.5| and |6.10[ there is a single 
bound that is proportional to \\x^~^^ — a;*||'*+^ that bounds each difference in (6.21). 

The second type of difference involves terms that are the difference between exact 
and computed values of derivatives with respect to the changes of coordinates of the 
optimal cost. These terms are 



dn 
dl 



(0) 



dw^ 



(0) 



k(0)-k'+^(0) 



1 



1 

1^ 



(6.22) 



By Theorem |4.H the difference between the computed and exact first order partial 
derivatives of the optimal cost is bounded by 



dir ^ 
dx 



dn 



i+l 



dx 



< T'Wx 



(6.23) 



for some T' < oo and by Lemma 6.5 there exists B < oo such that 



\V^V-I\\ < B\\x 



i+l 



(6.24) 



Each difference in (6.22 1 shares a single upper bound that is a linear combination 

\. The 

dn'+^ 



of the right hand sides of (6.23) and (6.24). The first difference is bounded by 

dTT'+^ 



dn 
dl 



(0) 



di 



-(0) 



dn 



[x 



dx 

<[T' + B sup 



dx 
dn 



-(x^+i) + g(x^+i)(Ff^-/) 



V 



dx 
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ix) 



„jllrf+l 



and the second difference obeys 



\\k{o)- k'+^{o)\\ < (r sup 



.9(^)11 



By assumption, the distance between a;' and x'^^ is less than the maximum consec- 
utive patch point distance h of CoroUary |4.2| , so 



sup 



oo 



(6.25) 



Xa\A 



and the third difference in (6.22) is bounded by 
1 



1 

Ji\\ 



< T' 



^ r(x)(||i||(||i||-||i-i^+i||)) 

\\x^ + ^-x'\\<h 



9i^)\\ 



Our next task is to demonstrate the contribution from the second order non 
characteristic partial derivatives to (6.17) is C'(||x*+"'^ — Let 7r*+-'^(^) = Tr{x^^^ + 

V^) where tt denotes the Taylor polynomial of the optimal cost centered at a; = x*. 
If neither ji nor j2 are one, then by Corollary |6.11[ the difference between the non 
characteristic partial derivatives is bounded by 



-(0) 



■J2 



(0) 



< 2" 



1) 



d 



d 



(2^ 



dx dx 

d_ d_ 
dx dx 



1 



■n{x) - tt{x)]x=x-+i (6.26) 



The first term on the right hand side of the inequality (6.261 is C'(||a;*"*'^ ~ x^W 



\d+i\ 



by the assumption that tt is smooth on the compact computational domain, and 
(6.24). Since tt is the Taylor polynomial of tt centered at x% then each entry in 
■§^® [7r(a;) — Tt(x)\x=x^+'^ is bounded by Rt\\x'''^'^ — a;*||'*, so the second term on the 
right hand side of the inequality in (6.26) is Odlx'^^ — a;'||''). 



We can conclude that the bound in (6.17) holds for A; = 2, so the conclusion of 



the theorem holds in the case of the second order partial derivatives of the optimal 
cost. 



By the formula for 



we can conclude that the difference between 



(0) in (2.13) and the same reasoning we just employed. 



2|^(0) and If (0) is 0{\\x^+^ 



The proof that the higher order partial derivatives satisfy the inequality in ( 6.17[ ) 
is analogous to the second order case. We outline the proof of the third order case, 
which is representative of the higher order cases. The computed third order charac- 
teristic partial derivatives with respect to the change of variables of the optimal cost 
is calculated from the formula in j ]9.2[ and the exact characteristic partial derivatives 
have an analogous formula. As in the case of the second order characteristic par- 
tial derivatives of the optimal cost, the bound on the difference between the exact 
and computed third order characteristic partial derivatives depends linearly on two 
types of differences. The first type of difference is the difference between the partial 
derivatives of problem data functions with respect to the two changes of variables, 
and each one of these differences is Odlx'^^ — The other type of difference 

33 



is the difference between the exact and computed partial derivatives with respect to 
the two changes of variables of the solution. These differences are 



1 

(0) 



1 



dk 



(0) 



dk'+^ 



(0) 



(6.27) 



Each one of these differences was treated in the second order case and is 0(110;*+^ — 

In the case of bounding the non characteristic partial derivatives, if none of the 
coordinate indices ji, j2, nor are one, then we may apply Corollary 6.11 to the 
difference between the non characteristic partial derivatives 



(0) 



OH 



(0) 
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< 2^ 



d d 

— (E) — <E 
dx dx 

d d 

— ® — 
dx dx 



dx 



dx 



[k{x) - 7r(x)], 



(6.28) 



„^||£i+l^ 



The first term on the right hand side of the inequality in (6.28) is 0(11 2^ 
by the assumption that tt is smooth on the compact computational domain, and 
(6.24). Since tt is the Taylor polynomial of tt centered at x*, then each entry in 
^^®'§x ® ~^{^)]x=xi+^ is bounded by Rt\\x^'^^ — so the second term 

on the right hand side of the inequality in (6.281 is 0(||x*+^ — a;*||''^^). 
□ 

7. Lipschitz condition for second and higher order partial derivatives. 

In this section we prove that (pj satisfies the Lipschitz condition when j > 2 under the 
conditions on the optimal control problem in §2.11 most importantly that the optimal 
cost is a strict Lyupanov function. 

Theorem 7.1. Under the conditions on the optimal control problem in %2.1\ there 
exists a maximum step size > 0, maximum multiplier Mmax > 0, and Lipschitz 
constant L2 such that for all x^ and x^^^ in Xc \ A within a distance of h^ of each 
other, and for all M < Mmax, 



\\MC'l~MCn\\ < L2M\\x 



il\d+2-k 



for2<k<d+l 



(7.1) 



whenever 



\C] - C]\\ < M\\x 



i+l 



r'\\d+2-j 



forO<j<d+l 



where C* is the vector of exact partial derivatives up to order d+l of the optimal cost 
at a;'. 

The proof mirrors the proof of the local truncation error result of Theorem |6.1| We 
will delay the proof of the theorem until after we have introduced some notation and 
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established some relevant bounds. We denote the two approximate optimal directions 
at 2;*+^ computed from and C' as 



and 

=fi^^ ) ^(^i^T) ) 

As in the case of the local truncation error result, we will work with two changes 
of variables that we choose to be convenient. These changes of coordinates are 

X = + V-. = jjflj (7.2) 

and 

a; = x'+^ + = (7.3) 

We denote the approximate optimal cost and optimal control polynomials under the 
two changes of variables as 

and we denote the problem data functions under the two changes of variables as 

fii) EE + vo Hi) ^ v^fix'+' + vi) 

q{0 ^ q{x'+' + vi) qii) ^ q{x'+' + H) 

r{i) EE r{x'+^ + vi) f(|) = r{x'+^ + f |) 



In the course of proving Theorem 7.1 we will need a guarantee that certain 
quantities depending on the computed solution are finite. Lemma |7.2| provides this 
guarantee. 

Lemma 7.2. If the optimal control problem satisfies the assumptions of ^2.1\ then 
there exists an M^ax > and a maximum consecutive patch point distance h2 > 
such that, for all M < Mmax md for all x^ and x^^^ in Xc \ A, 

inf||i;^+i|| > 

M{\\xl+'\\-\\xl+'^x^t'\\)>0 ^^'^^ 

whenever 

||a;*+i _ < h2 and ||q - q|| < Ad\\x'+^ - x'\\''+^-^ forO<j<d+l 
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where the infimums are taken over all M < Mmax and all x''^^ in Xc \ A such that 
x^^^ — a;* < /i2- C* holds the exact partial derivatives of it at a;* e \ A. 

Proof. The exact optimal direction at x^~^^ is x = /(x*^^) + g{x'^~^^)K{x''^^). 
holds the exact partial derivatives of the optimal cost at up to order d + 1, so by 



Theorem ^.1 



, lim . ((/(x^+i) + g{x'+^)K{x'+^)) - i^+i) = 0. (7.6) 

The assumption that it is a strict Lyapunov function guarantees that the infimum over 
Xc\A of \\x\\ is strictly greater than zero. The limit in (7.6) is uniform, so for all ei 
m < ei < 1, there exists a Si > independent of x'' and x"^^^ such that 

\\f{x^+') + g{x^+')K{x^+') - < 61 inf IIAII 

Therefore 

> \\f{x'+') + g{x^+'Hx^+')\\ - \\fix^+')+gix^+')K{x^+')~xl+^\\ 

> (1-ei) inf llxH 

Xa\A 

> 

Thus we have established the first inequality of the lemma statement. 

We will assume for the rest of the proof that M < Mmax- By Theorem \ 5.1\ if 
||C] - qil < M|lx*+i - x'f +2-^' forO<j<d+l, then 

lim xl+^-x'^^ =0. (7.7) 



The limit in (7.7) is uniform, so for all 62 in < £2 < 1; there exists a 62 > 
independent of and x'^^ such that 

\\xl'^'~-xf'\\<e2hd_\\x\\ 

Xa\A 

whenever 

- x'W < S2 and liq - qil < M\\x'+^ - x'\\'^+^-^ forQ<j<d+l. 
Therefore, if ||ic"^"'"|| > (1 — (1) inf ^ "where < ei < 1, then 

\\xl+'\\ - \\xl+' - > (1 - ei) JnLllill - ^2(1 - £1) inf HxH 

Xa\A Xa\A 

= (l-£i)(l-e2) inf IIAII 

Xc\A 

> 

whenever 

_ x'W < 61 and liq - qil < M\\x'+'^ - +2"^' forO<j<d+l 

so the second infimum of the lemma statement has been established. We take /12 to be 
the smaller of Si and 62 . □ 
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Corollary |7.3| provides a guarantee that any term depending on the difference 
between the normalized optimal directions at computed from C* and (7* does 
not affect the order of the main bound (7.1 1 in Theorem ] 7. 1| 

Corollary 7.3. If the optimal cost -k is a strict Lyapunov function on the com- 
putational domain Xc, then there exists a Lipschitz constant L' , maximum multiplier 
J^max > and a maximum consecutive patch point distance /12 > such that for all 
and x^'^^ in X^. \ A within a distance of /12 of each other, and for all M < M^^xi 



■r.i+1 



< L'Af|lx*+i - X 



whenever 



\\C)-C)\\ <M\\x 



i + l 



X 



i\\d+2-j 



forO<j<d+l 



where C* holds the exact partial derivatives of n at x^ £ Xc \ A. 

Proof. The bound in the corollary statement follows from the Lemma \7.^ and the 
inequality 



< 



ui+1 



As in the case of proving the local truncation error result of Theorem |6.1[ we will 



pick the changes of coordinates in (7.2 1 and (7.31 so that the orthogonal mat rices V 



7.4 



and V are almost inverses of each other. This is the content of Corollary 

Corollary 7.4. If the optimal cost -k is a strict Lyapunov function on the com- 
putational domain X^, then there exists a Lipschitz constant L' , maximum multiplier 
Mmax > and a maximum consecutive patch point distance h2 > 0, and orthogonal 
matrices V and V such that their first columns are 



so that for all and x''^^ in X^ \ A within a distance of of each other and all 

M < Mmax, 



|y^V^- /|| < L'M\\x 



whe 



iq-qii <M\\x'+'^^x 



i\\d+2-j 



forO<j<d+l 



where C* holds the exact partial derivatives of t: at x'' € Xc \ A, then 

Pr oof. For a sufficiently small maximum consecutive patch po int d istance /i2, 



Lemma 
laries 



7.2 guarantees x]^^ and ig^^ are close enough so that Lemma 
6.8 imply the hound on V'^V 



6. 1 and 



6.6, and Corol- 
I of the corollary statement. □ 
As in the proof of Theorem |6.1[ we bound the differences between the computed 
partial derivatives of the optimal cost with respect to the two changes of coordinates, 
and then show that the order of this bound does not change after recovering the two 
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sets of partial derivatives with respect to the original state space coordinates and 
compare their differences. Lemma |7. 5| provides a bound on the difference between the 
characteristic partial derivatives, which are computed by inheritance. 

Lemma 7.5. Suppose the orthogonal matrices V and V from the changes of 



coordinates in (7.2) and (7.3) are the ones from Corollary 7.4 If the optimal cost 



IT is a strict Lyapunov function on the computational domain X^, then there exists 
a Lipschitz constant L'^^f^ < oo, Mmax > and a maximum consecutive patch point 
distance /i2 > such that the partial derivatives computed by inheritance satisfy 



ik~i+l 



d TT 



-(0) 



i+1 _ ^i||d+2-fe 



3k 



whene 



||a;*+i - x'W < h2 and ||C; - q|| < A-I\\x'+^ - x'\\''+'^-^ forO<j<d+l 



Proof. P is linear in its first argument, and 



§- is linear, so by Lemma 



dx 



d_ 
dx 



dx 



[P{C\x-x'%^,,+^ 



d_ 
dx 



d_ 
dx 



[P{C\x-x%,-_ 



d 



d 

— (g) • • • ^ — 
dx dx 



where ^ 



is repeated k times. 



It follows from the local truncation error result of Theorems \4 . 1\ and \6.1\ that there 
is some T < oo such that 



d_ 
dx 



dx 



[P{C\ X - x') - tt{x) + tt{x)]^^^,- 



< T x'+' - X 



+ 1 _ i\\d+2-k 



sup 



d_ 
dx 



dx 



[TT{x)]^=^i+l 



where the supremum is finite since tt is smooth by assumption on the compact com- 



putational domain. It follows from Corollary \7.4\ and Lemma 6.9 that 



VV^ VV^ -I < (2'^' - l)L[M\\x'+^ - +1 



The conclusion of the lemma follows by applying the previous three inequalities to 
the following bound on the difference between the two computed characteristic partial 
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derivatives of the optimal cost. 



d TT 



(0) 



dx 



< 



d d 

— (g) • • ~ 
dx 

_ d_ 

dx 
d 

— ® ■ 
dx 

d_ 
dx 



[P{C\X%=,.^^V^^ 



yjk 



d_ 
dx 



yjk 



d_ 
dx 



[P{& -C\x)] 



d_ 
dx 



[P{C\x%^,.^.{VV^ 



VV'^ ~ I) 



□ 



Wc are now in a position to prove Theorem 7.1 



Proof. [Proof of Theorem 7.1 The proof of the theorem mirrors the proof of the 
local truncation error result in Theorem |6.1| We may assume that the orthogonal 



matrices V and V appearing in the changes of coordinates of (7.2 1 and (7.31 are 
convenient, that is they are taken from Corollary |7.4[ Starting with the computed 
second order partial derivatives, we will show that the main bound (7.1) of the theorem 
holds, by bounding the individual entries oi (t)k{C'^)—'Pk{C^)- We do this by considering 
the cases of the computed characteristic and non characteristic partial derivatives of 
the optimal cost independently, and then proceed by induction for the higher order 
partial derivatives. 

We denote the computed optimal cost polynomials computed from the coefficient 
sets C* and and centered at x*"^^ as 7r^+^ and tt^^^. They are given by the formulas 



■Kl+'ix) = Pi(t){C'),X - X'+') 



ix)^Picbin,x~x^+') 



At the difference between ap*^ order partial derivative of the two computed 
optimal cost polynomials is bounded by 



dx-j 



■ dx 



dP-rrl 



+1 



dxj-^ ■ ■ ■ dx 



d^ 

di 



VV^ -I 



d_ 

di 



9C 



d_ 

di 



5C 



(7.8) 



The following two inequalities will imply that the first term on the right hand side of 
(7.8) has an upper bound that is proportional to Af ||a;*+^ — a;'||'^+2-*: for any k > 2. 
Since 7r'+^ is computed from the set of coefhcients that are exact at the previous patch 
point, it follows from Theorem 6.1 that there is some T < oo such that 



d 

— ( 
d^ 



d^ 



< sup 

Xa\A 



d_ 
dx 



d_ 
dx 



[nix)] 



-T\\x 



i+l 



(7.9) 



By Corollary |7.4| and Lemma 6.9 



< {2'' 
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l)L[M\\x 



i+l 



^i||<i+l 



(7.10) 



We now prove that the second term on the right hand side of (7.8) has an upper 
bound that is proportional to Af||2;'+^ 



„i\\d+l 



. Each entry in 



d_ 



— = TT 
5^ 



i+1 



d_ 



d_ 



is either the difference between characteristic or non characteristic partial derivatives. 



In the case of the non characteristic partial derivatives, Lemma 7.5 guarantees that 
the difference is at most 



Qk^^+l 



(0) 



(0) <iUM||:r^+i-xl 



i\\d+2-k 



SO the non characteristic partials make a contribution to the right hand side of ( 7.8 1 
that is at most L[^^^M\\x^^^ — a;*||'*+2~'^. We now consider the contribution to (7.81 



from the characteristic partial derivatives. We start with the second order partial 
derivatives and proceed by induction. At x*"*"^, a characteristic partial derivative of 
the optimal cost is calculated from the coefficients held in from the formula 



(0) 



1 



i+1 1 



' ^^^^ 9^/ 



(0) 



2 9^ 



(7.11) 



and the corresponding characteristic partial derivative computed from the coefficients 
in are computed by an analogous formula. The task of showing that the differ- 
ence between the second order computed characteristic partial derivatives makes no 
more than an 0(M||a;*+^ — a;'||'^) contribution to the right hand side of (7.8) reduces 



to showing that the difference between the partial derivatives of the problem data 
functions with respect to the two changes of variables 



1(0) 



5/ 

dq 



(0) 



(0) 



dg 

di 

df 



(0)~S(0) 



df 



(7.12) 



dC ' dC dC ' 

and the difference between each of the computed quantities 



(0) 



i+1 



(0) 



are all proportional to M||x^_^ 
M\\x'+^ -x'\\'^+^. Lemma [elo 



(0) 



-j+i 



(0) 



1 



■J+ii 



(7.13) 



of the problem data functions in terms of V'^V 
for some Lipschitz constant L < oo 



- x^W^ or higher. In fact, they are all proportional to 
bounds the difference between the p artial derivatives 



/. Corollary 7.4 then implies that 



I"" 



dg 



1(0) 



< LMWx' 



and and the same bound bounds the other differences in (7.12). 
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We now show that there is some Lipschitz constant L > s uch that each difference 

there exists an L such 



that 



in ( [7l3| ) is bounded by LM\\x'+'^ - x'\\'^+^. By Theorem 

-(a;*+i) < LM\\x 



5.1 



dx 



dx 



and by the inequalities (7.9) and (7.10), there is some other L 



dx 



{x'+'){VV-'' -I) < LM\\x'+^ - x'f +1 



It then follows from 



^(o)-^(o) 



9C 



< 



dx 



dx 



0. <^-'">-^<^-"'> 



dx 



{x'+'){VV' - I) 



that for some L < oo, 



9C 



dTT'+^ 



(0) <LAf||2;^+i-a;''|l''+i 



The differences between the computed optimal controls is at most 



r{x^+^) 

< sup 

< sup M^iM|!a;'+i-x^f+i 



r(a:) 



Lemma 



7.2 



guarantees that for all and x*+^ in Xc \ A, 

inf ||i;^+i||(||x^+i||-||i^+i~i^+i||)>0 



whenever 



x'+' -x'W < h2 and \\Cl ~ Cl\\ < M\\x'+^ x'\\'^+^-'' for < fc < d + 1 



(7.14) 



(7.15) 



(7.16) 



Therefore, for some L, the final difference in (7.13) is bounded by 
1 



1 



< 



sup 1 1. g (a;) 1 1 (a;) 



inf(||i'.+i|j(|ji^+i||-||i^+i-il+i||)) 
< LM\\x' 



dx 



dx 



(7.17) 
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where the supremum is taken over Xc \ A, and the infimuni is taken over all and 
x'"'"^ in Xc.\A within a distance of of each other. Their ratio is then absorbed into 
L. 

The proof that the difference between every corresponding characteristic partial 
derivatives of order k > 2 shares a single upper bound that is proportional to M\\x''^^ — 
^i|j(i+2-fc proceeds by induction and follows the same form as the k = 2 case. 
□ 

8. Numerical results. We tested the patchy algorithm on the following non- 
linear test problem. 




subject to the dynamics 

Xl = (x2 - ^xl^ sec{xi) 

^2 = (xlx2 - ^Xi^ sec(xi) -I- u 

For any integer k, the optimal dynamics have a rest point at {xi,X2) = {kir, ^(/ctt)'^), 
and the optimal cost is zero, and the presence of sec(a;i) makes x blow up near 
Xl = — Tr/2 and xi — t:/2. 

Under a change of coordinates, the test problem is the linear-quadratic regulator 
problem 




subject to the dynamics 

m = 2/2 
y2=u 

and the change of variables is 

yi = sin(xi) 

1 3 (8.1) 

y2=X2 - -x^. 

We chose this nonlinear test problem because we can calculate the exact solution 
to the linear-quadratic regulator problem exactly by solving the associated algebraic 
Riccati matrix equation [7, p. 252]. We can then calculate the exact solution to the 
nonlinear test problem by applying the change of coordinates to the solution of the 
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linear-quadratic regulator problem. Doing so gives us the solution to the nonlinear 
optimal control problem 



/ N ^\( ■ , ^ ( 1 3^^^ 2 . 1 3 
7r(x) = — ^(^sm(2;i) + [x^ - -xj j + - (2:2 - ^2:^1 



The problem data functions /, q, and r and the solution tt are all smooth on 
{(x\,X2) I — 7r/2 < a;i < 7r/2, — 00 < 3:2 < 00}. Furthermore, tt is a strict Lyapunov 



function on this set. If we denote the change of variables (8.1 1 y = T{x), and if the 
matrix ^(x) is invertible at x, then ^{x)x reduces to the following quadratic form 
in T{x) 



-1 ■ 

V3 -2 



T{x), 



which is negative definite. Therefore, the computed solutions tt' obey the error bound 
of Theorem 13.61 

We computed the patchy solution on 73 patches including the Al'brekht patch. 
The maximum consecutive patch point distance was h « .54. The degree of the 
computed optimal cost and optimal control polynomials was four and three. To 
compute the absolute error, we created a 100 x 100 grid of equally spaced points in 
the square —1 < Xi,X2 < 1, and then computed the difference between the exact 
and computed optimal cost at each grid point that fell inside a patch. The maximum 
absolute error at a grid point was approximately 5 x 10^''. The theoretical absolute 
error bound grows along a sequence of consecutive patch points as K{^j-^ + l)h^ w 

K{j-^ + 1)(4.6 X 10~^). The computed and exact solutions are displayed in Figure 
8.1[ The absolute error is displayed in Figure [8?2| 

To verify the validity of the analytical error bound, we computed the patchy 
solution on five concentric level sets of the computed cost. We doubled the number 
of patch points on each level set, starting with eight points on the boundary of the 
Al'brekht patch. We then computed the absolute error of optimal cost at each patch 
point and then found the sequence of patch points that terminates with the greatest 
absolute error. The level sets of the optimal cost behave badly near the singularities 
at xi — ±7r/2, which limited the size of the computational domain for this test 
problem. In this case, /i w .5, so the absolute error bound grows as K{j-^ + l)h^ w 

K{ ^i^Zi + 1)(3-1 X 10~^). The absolute error and the sequence of patch points yielding 
the worst case error are shown in Figure [873} 
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Patchy Optimal Cost 




(a) Patchy Optimal Cost 



Exact Optimal Cost 




(b) Exact Optimal Cost 



Fig. 8.1. Patchy and exact optimal costs 
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Patch Points and Sequence of Greatest Error 



0.5 - 



-0.5 - 



-1 L 



* 



* 



* 



* 



patch poiiil 
^ max error sequeni; 



-1.5 -1 -0.5 0.5 1 1.5 

(a) Optimal cost absolute error at sequence of consecutive patch points. 




Fig. 8.3. Optimal cost absolute error at sequences of patch points, /i ~ .5 
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9. Appendix. 



9.1. Kronecker derivative notation. Definition 9.1. Let s : R" R^^™ 



S{x)=[si{x) S2{x) ■■■ Sm{x)] , 



The differential operator is -g^ ® s{x) is shorthand for the 1 x mn row vector 



dx 



I s{x) 



The higher order operator 
the special case where s : IB 



ax 



® ■ 



■ (El ^ is defined recursively in the natural way. In 
then If : M2 ^ M^^^^ and 



d_ 
dx 
d 

— (X) — 
dx dx 



d 



d d 



d 



dx dx dx 



Six)] = 
s{x)] ^ 



s{x)] ^ 



9.2. Partial derivative formulas for the optimal control and chetracter- 
istic partial derivatives of the optimal cost. 



(0) 



1 



(0)( 



+ 



^(0) + ^ 

df 



(0)K*+i(0)) 



+ 



(0)(^(0) + ^(0)«^+^(0)) 
(0)( 



(0) 
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^ (0) + (0)fe'+^(0) 



1 d'^f 



(0)k'+1(0)2 



(0) 



1 



(0)5(0) + ^^(0)^(0) 



+ 



(0)#(0) 



+ ^(0)-/^(0) 



(0)«'+^(0) 



(0)/c^+^(0) 



^(0)^(0) 



4-1+1 



1 



(0) = 



+ 



+ 



(0)( 



(0)) 



(0)( 



|^(0) + ^(0)K^+^(0)+^(0)^ 



%3 



-(0) 



'J2 



i2^i+l 



.(o)( 



5V 



-(0), 



(0) 
(0) 



(0)«^+i(0)) 
(0)K^+i(0)) 



+ 



IT 



(0)( 



+ 



Q2f 



-(0) 



(0) + 



-(0)(K*+i(0))^ 



-(0)«^+i (0)^^(0) + ^(0)^(0)^(0) 
d^ndi^. d^n 5^.3 5?.i ^^i. 

- ^(0) (0) (0) - r(0) (0) -^-^ (0) 



di 
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1 



(0) 



+ 



+ 



(0)5(0) 



(o)#-(o) + 



J3 



(0) 



5^ 



(0)^(0) 
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(0) ^ (0) + ^^(0) : \ (0) 



+ 



(0)^(0)- 



(0) 



(0)^^(0) 



Jl 



J2 



(0)4^^(0) 



^4 



f(0) 



(0) 
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